{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Pandas 处理缺失数据\n",
        "\n",
        "本教程详细介绍如何在 Pandas 中检测、删除和填充缺失数据（NaN、None）。\n",
        "\n",
        "## 目录\n",
        "1. 缺失值检测\n",
        "2. 删除缺失值\n",
        "3. 填充缺失值\n",
        "4. 插值方法\n",
        "5. 高级技巧"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 导入库"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {},
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 1. 缺失值检测\n",
        "\n",
        "### 方法说明\n",
        "\n",
        "Pandas 提供了多种方法来检测缺失值。\n",
        "\n",
        "**主要方法：**\n",
        "- `isna()` / `isnull()`: 检测缺失值，返回布尔 DataFrame\n",
        "- `notna()` / `notnull()`: 检测非缺失值，返回布尔 DataFrame\n",
        "- `isna().sum()`: 统计每列缺失值数量\n",
        "- `isna().mean()`: 统计每列缺失值比例\n",
        "\n",
        "**特点：**\n",
        "- ✅ `isna()` 和 `isnull()` 功能相同（别名）\n",
        "- ✅ 可以检测 `NaN`、`None`、`NaT`（时间缺失值）\n",
        "- ✅ 返回布尔值，便于筛选\n",
        "\n",
        "**适用场景：** 数据探索阶段，了解数据质量"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例1: 创建包含缺失值的数据"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "原始 DataFrame:\n",
            "     姓名    年龄       工资    部门       入职日期\n",
            "0    张三  25.0   8000.0    技术 2020-01-01\n",
            "1    李四  30.0      NaN    销售 2019-06-15\n",
            "2    王五   NaN  15000.0  None        NaT\n",
            "3    赵六  28.0      NaN    人事 2021-03-20\n",
            "4  None  32.0   9000.0    技术 2020-12-01\n",
            "\n",
            "数据类型:\n",
            "姓名              object\n",
            "年龄             float64\n",
            "工资             float64\n",
            "部门              object\n",
            "入职日期    datetime64[ns]\n",
            "dtype: object\n"
          ]
        }
      ],
      "source": [
        "df = pd.DataFrame({\n",
        "    '姓名': ['张三', '李四', '王五', '赵六', None],\n",
        "    '年龄': [25, 30, np.nan, 28, 32],\n",
        "    '工资': [8000, np.nan, 15000, np.nan, 9000],\n",
        "    '部门': ['技术', '销售', None, '人事', '技术'],\n",
        "    '入职日期': pd.to_datetime(['2020-01-01', '2019-06-15', None, '2021-03-20', '2020-12-01'])\n",
        "})\n",
        "\n",
        "print(\"原始 DataFrame:\")\n",
        "print(df)\n",
        "print(\"\\n数据类型:\")\n",
        "print(df.dtypes)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例2: 使用 isna() 检测缺失值\n",
        "\n",
        "`isna()` 返回布尔 DataFrame，True 表示缺失值。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "缺失值检测结果 (True表示缺失):\n",
            "      姓名     年龄     工资     部门   入职日期\n",
            "0  False  False  False  False  False\n",
            "1  False  False   True  False  False\n",
            "2  False   True  False   True   True\n",
            "3  False  False   True  False  False\n",
            "4   True  False  False  False  False\n",
            "\n",
            "或使用 isnull() (功能相同):\n",
            "      姓名     年龄     工资     部门   入职日期\n",
            "0  False  False  False  False  False\n",
            "1  False  False   True  False  False\n",
            "2  False   True  False   True   True\n",
            "3  False  False   True  False  False\n",
            "4   True  False  False  False  False\n"
          ]
        }
      ],
      "source": [
        "print(\"缺失值检测结果 (True表示缺失):\")\n",
        "print(df.isna())\n",
        "print(\"\\n或使用 isnull() (功能相同):\")\n",
        "print(df.isnull())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例3: 统计缺失值数量"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "每列缺失值数量:\n",
            "姓名      1\n",
            "年龄      1\n",
            "工资      2\n",
            "部门      1\n",
            "入职日期    1\n",
            "dtype: int64\n",
            "\n",
            "总缺失值数量:\n",
            "6\n"
          ]
        }
      ],
      "source": [
        "print(\"每列缺失值数量:\")\n",
        "print(df.isna().sum())\n",
        "print(\"\\n总缺失值数量:\")\n",
        "print(df.isna().sum().sum())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例4: 统计缺失值比例"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "每列缺失值比例:\n",
            "姓名      20.0%\n",
            "年龄      20.0%\n",
            "工资      40.0%\n",
            "部门      20.0%\n",
            "入职日期    20.0%\n",
            "dtype: object\n",
            "\n",
            "缺失值比例超过50%的列:\n",
            "姓名      20.0\n",
            "年龄      20.0\n",
            "工资      40.0\n",
            "部门      20.0\n",
            "入职日期    20.0\n",
            "dtype: float64\n"
          ]
        }
      ],
      "source": [
        "print(\"每列缺失值比例:\")\n",
        "missing_ratio = df.isna().mean() * 100\n",
        "print(missing_ratio.round(2).astype(str) + '%')\n",
        "print(\"\\n缺失值比例超过50%的列:\")\n",
        "print(missing_ratio[missing_ratio > 0.5])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### ⚠️ 重要理解：axis 参数的含义\n",
        "**axis 参数在不同操作中含义不同，这是一个常见困惑点：**\n",
        "\n",
        "#### 1. 在 `dropna(axis=0)` 中\n",
        "- `axis=0` 表示**操作的方向**：沿着行（row）的方向删除\n",
        "- 即：删除**整行**（删除包含缺失值的行）\n",
        "- `axis=1` 表示删除**整列**（删除包含缺失值的列）\n",
        "\n",
        "#### 2. 在 `isna().any(axis=1)` 中\n",
        "- `axis=1` 表示**检查的方向**：沿着列（column）的方向检查\n",
        "- 即：对每一**行**检查是否有任何**列**包含缺失值\n",
        "- `axis=0` 表示对每一**列**检查是否有任何**行**包含缺失值\n",
        "\n",
        "#### 记忆技巧：\n",
        "- **`dropna(axis=0)`**：axis=0 → 删除第 0 维（行） → 删除行\n",
        "- **`isna().any(axis=1)`**：axis=1 → 沿着第 1 维（列）聚合 → 检查每行\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "原始数据:\n",
            "     A    B   C\n",
            "0  1.0  5.0   9\n",
            "1  2.0  NaN  10\n",
            "2  NaN  7.0  11\n",
            "3  4.0  8.0  12\n",
            "\n",
            "缺失值分布:\n",
            "       A      B      C\n",
            "0  False  False  False\n",
            "1  False   True  False\n",
            "2   True  False  False\n",
            "3  False  False  False\n",
            "\n",
            "============================================================\n",
            "1. 在 dropna() 中的 axis:\n",
            "============================================================\n",
            "\n",
            "dropna(axis=0) - 删除行（默认）:\n",
            "沿着行方向操作，删除包含缺失值的整行\n",
            "     A    B   C\n",
            "0  1.0  5.0   9\n",
            "3  4.0  8.0  12\n",
            "\n",
            "dropna(axis=1) - 删除列:\n",
            "沿着列方向操作，删除包含缺失值的整列\n",
            "    C\n",
            "0   9\n",
            "1  10\n",
            "2  11\n",
            "3  12\n",
            "\n",
            "============================================================\n",
            "2. 在 isna().any() 中的 axis:\n",
            "============================================================\n",
            "\n",
            "isna().any(axis=0) - 检查每列:\n",
            "沿着行方向聚合，检查每列是否有任何行包含缺失值\n",
            "A     True\n",
            "B     True\n",
            "C    False\n",
            "dtype: bool\n",
            "\n",
            "包含缺失值的列:\n",
            "     A    B\n",
            "0  1.0  5.0\n",
            "1  2.0  NaN\n",
            "2  NaN  7.0\n",
            "3  4.0  8.0\n",
            "\n",
            "isna().any(axis=1) - 检查每行:\n",
            "沿着列方向聚合，检查每行是否有任何列包含缺失值\n",
            "0    False\n",
            "1     True\n",
            "2     True\n",
            "3    False\n",
            "dtype: bool\n",
            "\n",
            "包含缺失值的行:\n",
            "     A    B   C\n",
            "1  2.0  NaN  10\n",
            "2  NaN  7.0  11\n",
            "\n",
            "============================================================\n",
            "3. 对比总结:\n",
            "============================================================\n",
            "\n",
            "dropna(axis=0)     → 删除行（操作方向：删除整行）\n",
            "dropna(axis=1)     → 删除列（操作方向：删除整列）\n",
            "\n",
            "isna().any(axis=0) → 检查列（聚合方向：检查每列）\n",
            "isna().any(axis=1) → 检查行（聚合方向：检查每行）\n",
            "\n",
            "所以：\n",
            "- df.dropna(axis=0) 和 df[df.isna().any(axis=1)] 经常一起使用\n",
            "  → 先检查哪些行有缺失值，再删除这些行\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# 详细演示 axis 参数的区别\n",
        "\n",
        "# 创建示例数据\n",
        "df_demo = pd.DataFrame({\n",
        "    'A': [1, 2, np.nan, 4],\n",
        "    'B': [5, np.nan, 7, 8],\n",
        "    'C': [9, 10, 11, 12]\n",
        "})\n",
        "\n",
        "print(\"原始数据:\")\n",
        "print(df_demo)\n",
        "print(\"\\n缺失值分布:\")\n",
        "print(df_demo.isna())\n",
        "\n",
        "print(\"\\n\" + \"=\"*60)\n",
        "print(\"1. 在 dropna() 中的 axis:\")\n",
        "print(\"=\"*60)\n",
        "\n",
        "print(\"\\ndropna(axis=0) - 删除行（默认）:\")\n",
        "print(\"沿着行方向操作，删除包含缺失值的整行\")\n",
        "df_drop_rows = df_demo.dropna(axis=0)\n",
        "print(df_drop_rows)\n",
        "\n",
        "print(\"\\ndropna(axis=1) - 删除列:\")\n",
        "print(\"沿着列方向操作，删除包含缺失值的整列\")\n",
        "df_drop_cols = df_demo.dropna(axis=1)\n",
        "print(df_drop_cols)\n",
        "\n",
        "print(\"\\n\" + \"=\"*60)\n",
        "print(\"2. 在 isna().any() 中的 axis:\")\n",
        "print(\"=\"*60)\n",
        "\n",
        "print(\"\\nisna().any(axis=0) - 检查每列:\")\n",
        "print(\"沿着行方向聚合，检查每列是否有任何行包含缺失值\")\n",
        "col_has_na = df_demo.isna().any(axis=0)\n",
        "print(col_has_na)\n",
        "print(\"\\n包含缺失值的列:\")\n",
        "print(df_demo.loc[:, col_has_na])\n",
        "\n",
        "print(\"\\nisna().any(axis=1) - 检查每行:\")\n",
        "print(\"沿着列方向聚合，检查每行是否有任何列包含缺失值\")\n",
        "row_has_na = df_demo.isna().any(axis=1)\n",
        "print(row_has_na)\n",
        "print(\"\\n包含缺失值的行:\")\n",
        "print(df_demo[row_has_na])\n",
        "\n",
        "print(\"\\n\" + \"=\"*60)\n",
        "print(\"3. 对比总结:\")\n",
        "print(\"=\"*60)\n",
        "print(\"\"\"\n",
        "dropna(axis=0)     → 删除行（操作方向：删除整行）\n",
        "dropna(axis=1)     → 删除列（操作方向：删除整列）\n",
        "\n",
        "isna().any(axis=0) → 检查列（聚合方向：检查每列）\n",
        "isna().any(axis=1) → 检查行（聚合方向：检查每行）\n",
        "\n",
        "所以：\n",
        "- df.dropna(axis=0) 和 df[df.isna().any(axis=1)] 经常一起使用\n",
        "  → 先检查哪些行有缺失值，再删除这些行\n",
        "\"\"\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例5: 筛选包含缺失值的行"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "包含任何缺失值的行 (axis=1，按行判断):\n",
            "     姓名    年龄       工资    部门       入职日期\n",
            "1    李四  30.0      NaN    销售 2019-06-15\n",
            "2    王五   NaN  15000.0  None        NaT\n",
            "3    赵六  28.0      NaN    人事 2021-03-20\n",
            "4  None  32.0   9000.0    技术 2020-12-01\n",
            "\n",
            "包含全部缺失值的行 (axis=1，按行判断):\n",
            "Empty DataFrame\n",
            "Columns: [姓名, 年龄, 工资, 部门, 入职日期]\n",
            "Index: []\n"
          ]
        }
      ],
      "source": [
        "# axis=1 代表“按行判断”，即逐行检查每一行是否有缺失值\n",
        "# axis=0 代表“按列判断”，即逐列检查每一列是否有缺失值\n",
        "\n",
        "print(\"包含任何缺失值的行 (axis=1，按行判断):\")\n",
        "print(df[df.isna().any(axis=1)])\n",
        "\n",
        "print(\"\\n包含全部缺失值的行 (axis=1，按行判断):\")\n",
        "print(df[df.isna().all(axis=1)])\n",
        "\n",
        "# 示例：如果想判断包含任何缺失值的“列”，可以用 axis=0\n",
        "#print(\"包含任何缺失值的列 (axis=0，按列判断):\")\n",
        "#print(df.loc[:, df.isna().any(axis=0)])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例6: 使用 notna() 检测非缺失值\n",
        "\n",
        "`notna()` 与 `isna()` 相反，True 表示非缺失值。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "非缺失值检测结果 (True表示非缺失):\n",
            "      姓名     年龄     工资     部门   入职日期\n",
            "0   True   True   True   True   True\n",
            "1   True   True  False   True   True\n",
            "2   True  False   True  False  False\n",
            "3   True   True  False   True   True\n",
            "4  False   True   True   True   True\n",
            "\n",
            "筛选年龄非缺失的行:\n",
            "     姓名    年龄      工资  部门       入职日期\n",
            "0    张三  25.0  8000.0  技术 2020-01-01\n",
            "1    李四  30.0     NaN  销售 2019-06-15\n",
            "3    赵六  28.0     NaN  人事 2021-03-20\n",
            "4  None  32.0  9000.0  技术 2020-12-01\n"
          ]
        }
      ],
      "source": [
        "print(\"非缺失值检测结果 (True表示非缺失):\")\n",
        "print(df.notna())\n",
        "print(\"\\n筛选年龄非缺失的行:\")\n",
        "print(df[df['年龄'].notna()])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 2. 删除缺失值\n",
        "\n",
        "### 方法说明\n",
        "\n",
        "`dropna()` 用于删除包含缺失值的行或列。\n",
        "\n",
        "**语法:**\n",
        "```python\n",
        "df.dropna(axis=0, how='any', thresh=None, subset=None, inplace=False)\n",
        "```\n",
        "\n",
        "**主要参数：**\n",
        "- `axis`: 0 或 'index' 删除行，1 或 'columns' 删除列\n",
        "- `how`: 'any' 任何缺失就删除，'all' 全部缺失才删除\n",
        "- `thresh`: 非缺失值的最小数量\n",
        "- `subset`: 指定检查缺失值的列（仅对行操作有效）\n",
        "- `inplace`: 是否在原 DataFrame 上修改\n",
        "\n",
        "**特点：**\n",
        "- ✅ 可以删除行或列\n",
        "- ✅ 支持多种删除策略\n",
        "- ✅ 可以指定检查的列\n",
        "\n",
        "**适用场景：** 缺失值比例很小（<5%），或缺失值无法填充时"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例1: 创建示例数据"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "原始 DataFrame:\n",
            "   姓名    年龄       工资    部门\n",
            "0  张三  25.0   8000.0    技术\n",
            "1  李四  30.0      NaN    销售\n",
            "2  王五   NaN  15000.0    技术\n",
            "3  赵六  28.0      NaN    人事\n",
            "4  钱七   NaN   9000.0  None\n",
            "\n",
            "缺失值统计:\n",
            "姓名    0\n",
            "年龄    2\n",
            "工资    2\n",
            "部门    1\n",
            "dtype: int64\n"
          ]
        }
      ],
      "source": [
        "df = pd.DataFrame({\n",
        "    '姓名': ['张三', '李四', '王五', '赵六', '钱七'],\n",
        "    '年龄': [25, 30, np.nan, 28, np.nan],\n",
        "    '工资': [8000, np.nan, 15000, np.nan, 9000],\n",
        "    '部门': ['技术', '销售', '技术', '人事', None]\n",
        "})\n",
        "\n",
        "print(\"原始 DataFrame:\")\n",
        "print(df)\n",
        "print(\"\\n缺失值统计:\")\n",
        "print(df.isna().sum())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例2: 删除包含任何缺失值的行 (默认行为)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "删除包含任何缺失值的行 (how='any'):\n",
            "   姓名    年龄      工资  部门\n",
            "0  张三  25.0  8000.0  技术\n",
            "\n",
            "删除前: 5 行\n",
            "删除后: 1 行\n"
          ]
        }
      ],
      "source": [
        "df_cleaned = df.dropna()\n",
        "print(\"删除包含任何缺失值的行 (how='any'):\")\n",
        "print(df_cleaned)\n",
        "print(\"\\n删除前:\", len(df), \"行\")\n",
        "print(\"删除后:\", len(df_cleaned), \"行\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例3: 删除全部缺失的行"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "删除全部缺失的行 (how='all'):\n",
            "   姓名    年龄       工资    部门\n",
            "0  张三  25.0   8000.0    技术\n",
            "1  李四  30.0      NaN    销售\n",
            "2  王五   NaN  15000.0    技术\n",
            "3  赵六  28.0      NaN    人事\n",
            "4  钱七   NaN   9000.0  None\n",
            "\n",
            "删除前: 5 行\n",
            "删除后: 5 行\n"
          ]
        }
      ],
      "source": [
        "df_cleaned = df.dropna(how='all')\n",
        "print(\"删除全部缺失的行 (how='all'):\")\n",
        "print(df_cleaned)\n",
        "print(\"\\n删除前:\", len(df), \"行\")\n",
        "print(\"删除后:\", len(df_cleaned), \"行\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例4: 删除包含缺失值的列"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "删除包含缺失值的列 (axis=1):\n",
            "   姓名\n",
            "0  张三\n",
            "1  李四\n",
            "2  王五\n",
            "3  赵六\n",
            "4  钱七\n",
            "\n",
            "删除前: 4 列\n",
            "删除后: 1 列\n"
          ]
        }
      ],
      "source": [
        "df_cleaned = df.dropna(axis=1)\n",
        "print(\"删除包含缺失值的列 (axis=1):\")\n",
        "print(df_cleaned)\n",
        "print(\"\\n删除前:\", df.shape[1], \"列\")\n",
        "print(\"删除后:\", df_cleaned.shape[1], \"列\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例5: 使用 thresh 参数\n",
        "\n",
        "保留至少包含 N 个非缺失值的行。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "保留至少3个非缺失值的行 (thresh=3):\n",
            "   姓名    年龄       工资  部门\n",
            "0  张三  25.0   8000.0  技术\n",
            "1  李四  30.0      NaN  销售\n",
            "2  王五   NaN  15000.0  技术\n",
            "3  赵六  28.0      NaN  人事\n"
          ]
        }
      ],
      "source": [
        "df_cleaned = df.dropna(thresh=3)  # 保留至少3个非缺失值\n",
        "print(\"保留至少3个非缺失值的行 (thresh=3):\")\n",
        "print(df_cleaned)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例6: 指定检查的列 (subset)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "只检查'年龄'和'工资'列，删除这两列都有缺失的行:\n",
            "   姓名    年龄      工资  部门\n",
            "0  张三  25.0  8000.0  技术\n"
          ]
        }
      ],
      "source": [
        "df_cleaned = df.dropna(subset=['年龄', '工资'])\n",
        "print(\"只检查'年龄'和'工资'列，删除这两列都有缺失的行:\")\n",
        "print(df_cleaned)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例7: 使用 inplace 参数直接修改"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "使用 inplace=True 直接修改:\n",
            "   姓名    年龄      工资  部门\n",
            "0  张三  25.0  8000.0  技术\n"
          ]
        }
      ],
      "source": [
        "df_copy = df.copy()\n",
        "df_copy.dropna(inplace=True)\n",
        "print(\"使用 inplace=True 直接修改:\")\n",
        "print(df_copy)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 3. 填充缺失值\n",
        "\n",
        "### 方法说明\n",
        "\n",
        "`fillna()` 用于填充缺失值。\n",
        "\n",
        "**语法:**\n",
        "```python\n",
        "df.fillna(value=None, method=None, axis=None, inplace=False, limit=None)\n",
        "```\n",
        "\n",
        "**主要参数：**\n",
        "- `value`: 填充值（标量、字典、Series、DataFrame）\n",
        "- `method`: 填充方法 'ffill'（前向填充）、'bfill'（后向填充）\n",
        "- `axis`: 填充方向（0 行，1 列）\n",
        "- `limit`: 连续填充的最大数量\n",
        "- `inplace`: 是否在原 DataFrame 上修改\n",
        "\n",
        "**特点：**\n",
        "- ✅ 支持多种填充策略\n",
        "- ✅ 可以按列指定不同的填充值\n",
        "- ✅ 支持前向/后向填充\n",
        "\n",
        "**适用场景：** 缺失值可以合理填充时"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例1: 创建示例数据"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "原始 DataFrame:\n",
            "   姓名    年龄       工资    部门\n",
            "0  张三  25.0   8000.0    技术\n",
            "1  李四  30.0      NaN    销售\n",
            "2  王五   NaN  15000.0  None\n",
            "3  赵六  28.0      NaN    人事\n",
            "4  钱七   NaN   9000.0  None\n"
          ]
        }
      ],
      "source": [
        "df = pd.DataFrame({\n",
        "    '姓名': ['张三', '李四', '王五', '赵六', '钱七'],\n",
        "    '年龄': [25, 30, np.nan, 28, np.nan],\n",
        "    '工资': [8000, np.nan, 15000, np.nan, 9000],\n",
        "    '部门': ['技术', '销售', None, '人事', None]\n",
        "})\n",
        "\n",
        "print(\"原始 DataFrame:\")\n",
        "print(df)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例2: 使用常数填充"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "使用0填充数值列:\n",
            "   姓名    年龄       工资  部门\n",
            "0  张三  25.0   8000.0  技术\n",
            "1  李四  30.0      0.0  销售\n",
            "2  王五   0.0  15000.0   0\n",
            "3  赵六  28.0      0.0  人事\n",
            "4  钱七   0.0   9000.0   0\n",
            "\n",
            "使用'未知'填充字符串列:\n",
            "   姓名    年龄       工资  部门\n",
            "0  张三  25.0   8000.0  技术\n",
            "1  李四  30.0       未知  销售\n",
            "2  王五    未知  15000.0  未知\n",
            "3  赵六  28.0       未知  人事\n",
            "4  钱七    未知   9000.0  未知\n"
          ]
        }
      ],
      "source": [
        "print(\"使用0填充数值列:\")\n",
        "df_filled = df.fillna(0)\n",
        "print(df_filled)\n",
        "\n",
        "print(\"\\n使用'未知'填充字符串列:\")\n",
        "df_filled = df.fillna('未知')\n",
        "print(df_filled)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例3: 使用字典按列指定填充值"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "按列指定填充值:\n",
            "   姓名         年龄       工资   部门\n",
            "0  张三  25.000000   8000.0   技术\n",
            "1  李四  30.000000   9000.0   销售\n",
            "2  王五  27.666667  15000.0  未分配\n",
            "3  赵六  28.000000   9000.0   人事\n",
            "4  钱七  27.666667   9000.0  未分配\n"
          ]
        }
      ],
      "source": [
        "fill_values = {\n",
        "    '年龄': df['年龄'].mean(),  # 用均值填充\n",
        "    '工资': df['工资'].median(),  # 用中位数填充\n",
        "    '部门': '未分配'  # 用字符串填充\n",
        "}\n",
        "\n",
        "df_filled = df.fillna(fill_values)\n",
        "print(\"按列指定填充值:\")\n",
        "print(df_filled)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例4: 使用统计值填充"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "使用均值填充年龄:\n",
            "   姓名         年龄       工资    部门\n",
            "0  张三  25.000000   8000.0    技术\n",
            "1  李四  30.000000      NaN    销售\n",
            "2  王五  27.666667  15000.0  None\n",
            "3  赵六  28.000000      NaN    人事\n",
            "4  钱七  27.666667   9000.0  None\n",
            "\n",
            "使用中位数填充工资 (对异常值更稳健):\n",
            "   姓名    年龄       工资    部门\n",
            "0  张三  25.0   8000.0    技术\n",
            "1  李四  30.0   9000.0    销售\n",
            "2  王五   NaN  15000.0  None\n",
            "3  赵六  28.0   9000.0    人事\n",
            "4  钱七   NaN   9000.0  None\n",
            "\n",
            "使用众数填充部门:\n",
            "   姓名    年龄       工资  部门\n",
            "0  张三  25.0   8000.0  技术\n",
            "1  李四  30.0      NaN  销售\n",
            "2  王五   NaN  15000.0  人事\n",
            "3  赵六  28.0      NaN  人事\n",
            "4  钱七   NaN   9000.0  人事\n"
          ]
        }
      ],
      "source": [
        "print(\"使用均值填充年龄:\")\n",
        "df_filled = df.copy()\n",
        "df_filled['年龄'] = df_filled['年龄'].fillna(df_filled['年龄'].mean())\n",
        "print(df_filled)\n",
        "\n",
        "print(\"\\n使用中位数填充工资 (对异常值更稳健):\")\n",
        "df_filled = df.copy()\n",
        "df_filled['工资'] = df_filled['工资'].fillna(df_filled['工资'].median())\n",
        "print(df_filled)\n",
        "\n",
        "print(\"\\n使用众数填充部门:\")\n",
        "df_filled = df.copy()\n",
        "df_filled['部门'] = df_filled['部门'].fillna(df_filled['部门'].mode()[0] if not df_filled['部门'].mode().empty else '未知')\n",
        "print(df_filled)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例5: 前向填充 (forward fill, ffill)\n",
        "\n",
        "用前面的值填充。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "原始数据:\n",
            "          日期    温度    湿度\n",
            "0 2024-01-01  20.0  60.0\n",
            "1 2024-01-02  22.0   NaN\n",
            "2 2024-01-03   NaN  65.0\n",
            "3 2024-01-04   NaN  66.0\n",
            "4 2024-01-05  25.0   NaN\n",
            "5 2024-01-06  26.0  70.0\n",
            "6 2024-01-07   NaN  68.0\n",
            "\n",
            "前向填充 (method='ffill'):\n",
            "          日期    温度    湿度\n",
            "0 2024-01-01  20.0  60.0\n",
            "1 2024-01-02  22.0  60.0\n",
            "2 2024-01-03  22.0  65.0\n",
            "3 2024-01-04  22.0  66.0\n",
            "4 2024-01-05  25.0  66.0\n",
            "5 2024-01-06  26.0  70.0\n",
            "6 2024-01-07  26.0  68.0\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/var/folders/9b/jb3nnnqn60x3dtlll6mcf0000000gp/T/ipykernel_92897/577627238.py:11: FutureWarning: DataFrame.fillna with 'method' is deprecated and will raise in a future version. Use obj.ffill() or obj.bfill() instead.\n",
            "  df_ffill = df_time.fillna(method='ffill')\n"
          ]
        }
      ],
      "source": [
        "df_time = pd.DataFrame({\n",
        "    '日期': pd.date_range('2024-01-01', periods=7),\n",
        "    '温度': [20, 22, np.nan, np.nan, 25, 26, np.nan],\n",
        "    '湿度': [60, np.nan, 65, 66, np.nan, 70, 68]\n",
        "})\n",
        "\n",
        "print(\"原始数据:\")\n",
        "print(df_time)\n",
        "\n",
        "print(\"\\n前向填充 (method='ffill'):\")\n",
        "df_ffill = df_time.fillna(method='ffill')\n",
        "print(df_ffill)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例6: 后向填充 (backward fill, bfill)\n",
        "\n",
        "用后面的值填充。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "原始数据:\n",
            "          日期    温度    湿度\n",
            "0 2024-01-01  20.0  60.0\n",
            "1 2024-01-02  22.0   NaN\n",
            "2 2024-01-03   NaN  65.0\n",
            "3 2024-01-04   NaN  66.0\n",
            "4 2024-01-05  25.0   NaN\n",
            "5 2024-01-06  26.0  70.0\n",
            "6 2024-01-07   NaN  68.0\n",
            "\n",
            "后向填充 (method='bfill'):\n",
            "          日期    温度    湿度\n",
            "0 2024-01-01  20.0  60.0\n",
            "1 2024-01-02  22.0  65.0\n",
            "2 2024-01-03  25.0  65.0\n",
            "3 2024-01-04  25.0  66.0\n",
            "4 2024-01-05  25.0  70.0\n",
            "5 2024-01-06  26.0  70.0\n",
            "6 2024-01-07   NaN  68.0\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/var/folders/9b/jb3nnnqn60x3dtlll6mcf0000000gp/T/ipykernel_92897/3211527652.py:5: FutureWarning: DataFrame.fillna with 'method' is deprecated and will raise in a future version. Use obj.ffill() or obj.bfill() instead.\n",
            "  df_bfill = df_time.fillna(method='bfill')\n"
          ]
        }
      ],
      "source": [
        "print(\"原始数据:\")\n",
        "print(df_time)\n",
        "\n",
        "print(\"\\n后向填充 (method='bfill'):\")\n",
        "df_bfill = df_time.fillna(method='bfill')\n",
        "print(df_bfill)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例7: 限制连续填充数量"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "原始数据:\n",
            "          日期    温度    湿度\n",
            "0 2024-01-01  20.0  60.0\n",
            "1 2024-01-02  22.0   NaN\n",
            "2 2024-01-03   NaN  65.0\n",
            "3 2024-01-04   NaN  66.0\n",
            "4 2024-01-05  25.0   NaN\n",
            "5 2024-01-06  26.0  70.0\n",
            "6 2024-01-07   NaN  68.0\n",
            "\n",
            "前向填充，最多连续填充1个 (limit=1):\n",
            "          日期    温度    湿度\n",
            "0 2024-01-01  20.0  60.0\n",
            "1 2024-01-02  22.0  60.0\n",
            "2 2024-01-03  22.0  65.0\n",
            "3 2024-01-04   NaN  66.0\n",
            "4 2024-01-05  25.0  66.0\n",
            "5 2024-01-06  26.0  70.0\n",
            "6 2024-01-07  26.0  68.0\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/var/folders/9b/jb3nnnqn60x3dtlll6mcf0000000gp/T/ipykernel_92897/2963802386.py:5: FutureWarning: DataFrame.fillna with 'method' is deprecated and will raise in a future version. Use obj.ffill() or obj.bfill() instead.\n",
            "  df_limited = df_time.fillna(method='ffill', limit=1)\n"
          ]
        }
      ],
      "source": [
        "print(\"原始数据:\")\n",
        "print(df_time)\n",
        "\n",
        "print(\"\\n前向填充，最多连续填充1个 (limit=1):\")\n",
        "df_limited = df_time.fillna(method='ffill', limit=1)\n",
        "print(df_limited)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 4. 插值方法\n",
        "\n",
        "### 方法说明\n",
        "\n",
        "`interpolate()` 使用插值方法填充缺失值。\n",
        "\n",
        "**语法:**\n",
        "```python\n",
        "df.interpolate(method='linear', axis=0, limit=None, inplace=False)\n",
        "```\n",
        "\n",
        "**主要方法：**\n",
        "- 'linear': 线性插值（默认）\n",
        "- 'polynomial': 多项式插值\n",
        "- 'spline': 样条插值\n",
        "- 'time': 时间序列插值\n",
        "\n",
        "**特点：**\n",
        "- ✅ 比简单填充更精确\n",
        "- ✅ 适合连续数值数据\n",
        "- ✅ 考虑了数据趋势\n",
        "\n",
        "**适用场景：** 时间序列数据或有序数值数据"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例1: 线性插值"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "原始数据:\n",
            "   x     y\n",
            "0  1  10.0\n",
            "1  2   NaN\n",
            "2  3  30.0\n",
            "3  4   NaN\n",
            "4  5   NaN\n",
            "5  6  60.0\n",
            "6  7  70.0\n",
            "\n",
            "线性插值:\n",
            "   x     y\n",
            "0  1  10.0\n",
            "1  2  20.0\n",
            "2  3  30.0\n",
            "3  4  40.0\n",
            "4  5  50.0\n",
            "5  6  60.0\n",
            "6  7  70.0\n"
          ]
        }
      ],
      "source": [
        "df_linear = pd.DataFrame({\n",
        "    'x': [1, 2, 3, 4, 5, 6, 7],\n",
        "    'y': [10, np.nan, 30, np.nan, np.nan, 60, 70]\n",
        "})\n",
        "\n",
        "print(\"原始数据:\")\n",
        "print(df_linear)\n",
        "\n",
        "print(\"\\n线性插值:\")\n",
        "df_linear['y'] = df_linear['y'].interpolate()\n",
        "print(df_linear)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例2: 时间序列插值"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "原始数据:\n",
            "          日期    销售额\n",
            "0 2024-01-01  100.0\n",
            "1 2024-01-02  120.0\n",
            "2 2024-01-03    NaN\n",
            "3 2024-01-04    NaN\n",
            "4 2024-01-05  180.0\n",
            "5 2024-01-06  200.0\n",
            "6 2024-01-07  220.0\n",
            "\n",
            "时间序列插值:\n"
          ]
        },
        {
          "ename": "ValueError",
          "evalue": "time-weighted interpolation only works on Series or DataFrames with a DatetimeIndex",
          "output_type": "error",
          "traceback": [
            "\u001b[31m---------------------------------------------------------------------------\u001b[39m",
            "\u001b[31mValueError\u001b[39m                                Traceback (most recent call last)",
            "\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[32]\u001b[39m\u001b[32m, line 10\u001b[39m\n\u001b[32m      7\u001b[39m \u001b[38;5;28mprint\u001b[39m(df_time)\n\u001b[32m      9\u001b[39m \u001b[38;5;28mprint\u001b[39m(\u001b[33m\"\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[33m时间序列插值:\u001b[39m\u001b[33m\"\u001b[39m)\n\u001b[32m---> \u001b[39m\u001b[32m10\u001b[39m df_time[\u001b[33m'\u001b[39m\u001b[33m销售额\u001b[39m\u001b[33m'\u001b[39m] = \u001b[43mdf_time\u001b[49m\u001b[43m[\u001b[49m\u001b[33;43m'\u001b[39;49m\u001b[33;43m销售额\u001b[39;49m\u001b[33;43m'\u001b[39;49m\u001b[43m]\u001b[49m\u001b[43m.\u001b[49m\u001b[43minterpolate\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmethod\u001b[49m\u001b[43m=\u001b[49m\u001b[33;43m'\u001b[39;49m\u001b[33;43mtime\u001b[39;49m\u001b[33;43m'\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[32m     11\u001b[39m \u001b[38;5;28mprint\u001b[39m(df_time)\n",
            "\u001b[36mFile \u001b[39m\u001b[32m/opt/anaconda3/envs/ml311/lib/python3.11/site-packages/pandas/core/generic.py:8526\u001b[39m, in \u001b[36mNDFrame.interpolate\u001b[39m\u001b[34m(self, method, axis, limit, inplace, limit_direction, limit_area, downcast, **kwargs)\u001b[39m\n\u001b[32m   8524\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[32m   8525\u001b[39m     index = missing.get_interp_index(method, obj.index)\n\u001b[32m-> \u001b[39m\u001b[32m8526\u001b[39m     new_data = \u001b[43mobj\u001b[49m\u001b[43m.\u001b[49m\u001b[43m_mgr\u001b[49m\u001b[43m.\u001b[49m\u001b[43minterpolate\u001b[49m\u001b[43m(\u001b[49m\n\u001b[32m   8527\u001b[39m \u001b[43m        \u001b[49m\u001b[43mmethod\u001b[49m\u001b[43m=\u001b[49m\u001b[43mmethod\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   8528\u001b[39m \u001b[43m        \u001b[49m\u001b[43mindex\u001b[49m\u001b[43m=\u001b[49m\u001b[43mindex\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   8529\u001b[39m \u001b[43m        \u001b[49m\u001b[43mlimit\u001b[49m\u001b[43m=\u001b[49m\u001b[43mlimit\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   8530\u001b[39m \u001b[43m        \u001b[49m\u001b[43mlimit_direction\u001b[49m\u001b[43m=\u001b[49m\u001b[43mlimit_direction\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   8531\u001b[39m \u001b[43m        \u001b[49m\u001b[43mlimit_area\u001b[49m\u001b[43m=\u001b[49m\u001b[43mlimit_area\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   8532\u001b[39m \u001b[43m        \u001b[49m\u001b[43minplace\u001b[49m\u001b[43m=\u001b[49m\u001b[43minplace\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   8533\u001b[39m \u001b[43m        \u001b[49m\u001b[43mdowncast\u001b[49m\u001b[43m=\u001b[49m\u001b[43mdowncast\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   8534\u001b[39m \u001b[43m        \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   8535\u001b[39m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m   8537\u001b[39m result = \u001b[38;5;28mself\u001b[39m._constructor_from_mgr(new_data, axes=new_data.axes)\n\u001b[32m   8538\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m should_transpose:\n",
            "\u001b[36mFile \u001b[39m\u001b[32m/opt/anaconda3/envs/ml311/lib/python3.11/site-packages/pandas/core/internals/base.py:291\u001b[39m, in \u001b[36mDataManager.interpolate\u001b[39m\u001b[34m(self, inplace, **kwargs)\u001b[39m\n\u001b[32m    290\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34minterpolate\u001b[39m(\u001b[38;5;28mself\u001b[39m, inplace: \u001b[38;5;28mbool\u001b[39m, **kwargs) -> Self:\n\u001b[32m--> \u001b[39m\u001b[32m291\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mapply_with_block\u001b[49m\u001b[43m(\u001b[49m\n\u001b[32m    292\u001b[39m \u001b[43m        \u001b[49m\u001b[33;43m\"\u001b[39;49m\u001b[33;43minterpolate\u001b[39;49m\u001b[33;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[32m    293\u001b[39m \u001b[43m        \u001b[49m\u001b[43minplace\u001b[49m\u001b[43m=\u001b[49m\u001b[43minplace\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    294\u001b[39m \u001b[43m        \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    295\u001b[39m \u001b[43m        \u001b[49m\u001b[43musing_cow\u001b[49m\u001b[43m=\u001b[49m\u001b[43musing_copy_on_write\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    296\u001b[39m \u001b[43m        \u001b[49m\u001b[43malready_warned\u001b[49m\u001b[43m=\u001b[49m\u001b[43m_AlreadyWarned\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    297\u001b[39m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n",
            "\u001b[36mFile \u001b[39m\u001b[32m/opt/anaconda3/envs/ml311/lib/python3.11/site-packages/pandas/core/internals/managers.py:363\u001b[39m, in \u001b[36mBaseBlockManager.apply\u001b[39m\u001b[34m(self, f, align_keys, **kwargs)\u001b[39m\n\u001b[32m    361\u001b[39m         applied = b.apply(f, **kwargs)\n\u001b[32m    362\u001b[39m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[32m--> \u001b[39m\u001b[32m363\u001b[39m         applied = \u001b[38;5;28;43mgetattr\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mb\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mf\u001b[49m\u001b[43m)\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m    364\u001b[39m     result_blocks = extend_blocks(applied, result_blocks)\n\u001b[32m    366\u001b[39m out = \u001b[38;5;28mtype\u001b[39m(\u001b[38;5;28mself\u001b[39m).from_blocks(result_blocks, \u001b[38;5;28mself\u001b[39m.axes)\n",
            "\u001b[36mFile \u001b[39m\u001b[32m/opt/anaconda3/envs/ml311/lib/python3.11/site-packages/pandas/core/internals/blocks.py:1863\u001b[39m, in \u001b[36mBlock.interpolate\u001b[39m\u001b[34m(self, method, index, inplace, limit, limit_direction, limit_area, downcast, using_cow, already_warned, **kwargs)\u001b[39m\n\u001b[32m   1860\u001b[39m copy, refs = \u001b[38;5;28mself\u001b[39m._get_refs_and_copy(using_cow, inplace)\n\u001b[32m   1862\u001b[39m \u001b[38;5;66;03m# Dispatch to the EA method.\u001b[39;00m\n\u001b[32m-> \u001b[39m\u001b[32m1863\u001b[39m new_values = \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43marray_values\u001b[49m\u001b[43m.\u001b[49m\u001b[43minterpolate\u001b[49m\u001b[43m(\u001b[49m\n\u001b[32m   1864\u001b[39m \u001b[43m    \u001b[49m\u001b[43mmethod\u001b[49m\u001b[43m=\u001b[49m\u001b[43mmethod\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   1865\u001b[39m \u001b[43m    \u001b[49m\u001b[43maxis\u001b[49m\u001b[43m=\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mndim\u001b[49m\u001b[43m \u001b[49m\u001b[43m-\u001b[49m\u001b[43m \u001b[49m\u001b[32;43m1\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[32m   1866\u001b[39m \u001b[43m    \u001b[49m\u001b[43mindex\u001b[49m\u001b[43m=\u001b[49m\u001b[43mindex\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   1867\u001b[39m \u001b[43m    \u001b[49m\u001b[43mlimit\u001b[49m\u001b[43m=\u001b[49m\u001b[43mlimit\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   1868\u001b[39m \u001b[43m    \u001b[49m\u001b[43mlimit_direction\u001b[49m\u001b[43m=\u001b[49m\u001b[43mlimit_direction\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   1869\u001b[39m \u001b[43m    \u001b[49m\u001b[43mlimit_area\u001b[49m\u001b[43m=\u001b[49m\u001b[43mlimit_area\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   1870\u001b[39m \u001b[43m    \u001b[49m\u001b[43mcopy\u001b[49m\u001b[43m=\u001b[49m\u001b[43mcopy\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   1871\u001b[39m \u001b[43m    \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   1872\u001b[39m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m   1873\u001b[39m data = extract_array(new_values, extract_numpy=\u001b[38;5;28;01mTrue\u001b[39;00m)\n\u001b[32m   1875\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m (\n\u001b[32m   1876\u001b[39m     \u001b[38;5;129;01mnot\u001b[39;00m copy\n\u001b[32m   1877\u001b[39m     \u001b[38;5;129;01mand\u001b[39;00m warn_copy_on_write()\n\u001b[32m   1878\u001b[39m     \u001b[38;5;129;01mand\u001b[39;00m already_warned \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[32m   1879\u001b[39m     \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m already_warned.warned_already\n\u001b[32m   1880\u001b[39m ):\n",
            "\u001b[36mFile \u001b[39m\u001b[32m/opt/anaconda3/envs/ml311/lib/python3.11/site-packages/pandas/core/arrays/numpy_.py:310\u001b[39m, in \u001b[36mNumpyExtensionArray.interpolate\u001b[39m\u001b[34m(self, method, axis, index, limit, limit_direction, limit_area, copy, **kwargs)\u001b[39m\n\u001b[32m    307\u001b[39m     out_data = \u001b[38;5;28mself\u001b[39m._ndarray.copy()\n\u001b[32m    309\u001b[39m \u001b[38;5;66;03m# TODO: assert we have floating dtype?\u001b[39;00m\n\u001b[32m--> \u001b[39m\u001b[32m310\u001b[39m \u001b[43mmissing\u001b[49m\u001b[43m.\u001b[49m\u001b[43minterpolate_2d_inplace\u001b[49m\u001b[43m(\u001b[49m\n\u001b[32m    311\u001b[39m \u001b[43m    \u001b[49m\u001b[43mout_data\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    312\u001b[39m \u001b[43m    \u001b[49m\u001b[43mmethod\u001b[49m\u001b[43m=\u001b[49m\u001b[43mmethod\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    313\u001b[39m \u001b[43m    \u001b[49m\u001b[43maxis\u001b[49m\u001b[43m=\u001b[49m\u001b[43maxis\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    314\u001b[39m \u001b[43m    \u001b[49m\u001b[43mindex\u001b[49m\u001b[43m=\u001b[49m\u001b[43mindex\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    315\u001b[39m \u001b[43m    \u001b[49m\u001b[43mlimit\u001b[49m\u001b[43m=\u001b[49m\u001b[43mlimit\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    316\u001b[39m \u001b[43m    \u001b[49m\u001b[43mlimit_direction\u001b[49m\u001b[43m=\u001b[49m\u001b[43mlimit_direction\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    317\u001b[39m \u001b[43m    \u001b[49m\u001b[43mlimit_area\u001b[49m\u001b[43m=\u001b[49m\u001b[43mlimit_area\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    318\u001b[39m \u001b[43m    \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    319\u001b[39m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m    320\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m copy:\n\u001b[32m    321\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\n",
            "\u001b[36mFile \u001b[39m\u001b[32m/opt/anaconda3/envs/ml311/lib/python3.11/site-packages/pandas/core/missing.py:373\u001b[39m, in \u001b[36minterpolate_2d_inplace\u001b[39m\u001b[34m(data, index, axis, method, limit, limit_direction, limit_area, fill_value, mask, **kwargs)\u001b[39m\n\u001b[32m    371\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m method == \u001b[33m\"\u001b[39m\u001b[33mtime\u001b[39m\u001b[33m\"\u001b[39m:\n\u001b[32m    372\u001b[39m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m needs_i8_conversion(index.dtype):\n\u001b[32m--> \u001b[39m\u001b[32m373\u001b[39m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[32m    374\u001b[39m             \u001b[33m\"\u001b[39m\u001b[33mtime-weighted interpolation only works \u001b[39m\u001b[33m\"\u001b[39m\n\u001b[32m    375\u001b[39m             \u001b[33m\"\u001b[39m\u001b[33mon Series or DataFrames with a \u001b[39m\u001b[33m\"\u001b[39m\n\u001b[32m    376\u001b[39m             \u001b[33m\"\u001b[39m\u001b[33mDatetimeIndex\u001b[39m\u001b[33m\"\u001b[39m\n\u001b[32m    377\u001b[39m         )\n\u001b[32m    378\u001b[39m     method = \u001b[33m\"\u001b[39m\u001b[33mvalues\u001b[39m\u001b[33m\"\u001b[39m\n\u001b[32m    380\u001b[39m limit_direction = validate_limit_direction(limit_direction)\n",
            "\u001b[31mValueError\u001b[39m: time-weighted interpolation only works on Series or DataFrames with a DatetimeIndex"
          ]
        }
      ],
      "source": [
        "df_time = pd.DataFrame({\n",
        "    '日期': pd.date_range('2024-01-01', periods=7, freq='D'),\n",
        "    '销售额': [100, 120, np.nan, np.nan, 180, 200, 220]\n",
        "})\n",
        "\n",
        "print(\"原始数据:\")\n",
        "print(df_time)\n",
        "\n",
        "print(\"\\n时间序列插值:\")\n",
        "df_time['销售额'] = df_time['销售额'].interpolate(method='time')\n",
        "print(df_time)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 5. 高级技巧\n",
        "\n",
        "### 技巧1: 组合使用 fillna 和 groupby\n",
        "\n",
        "按分组填充缺失值。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "df = pd.DataFrame({\n",
        "    '部门': ['技术', '技术', '销售', '销售', '人事', '人事'],\n",
        "    '姓名': ['张三', '李四', '王五', '赵六', '钱七', '孙八'],\n",
        "    '工资': [8000, np.nan, 12000, np.nan, 6000, np.nan]\n",
        "})\n",
        "\n",
        "print(\"原始数据:\")\n",
        "print(df)\n",
        "\n",
        "print(\"\\n按部门均值填充:\")\n",
        "df['工资'] = df.groupby('部门')['工资'].transform(lambda x: x.fillna(x.mean()))\n",
        "print(df)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 技巧2: 创建缺失值指示变量\n",
        "\n",
        "有时缺失本身是有意义的，可以创建一个指示变量。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "df = pd.DataFrame({\n",
        "    '姓名': ['张三', '李四', '王五', '赵六'],\n",
        "    '年龄': [25, 30, np.nan, 28]\n",
        "})\n",
        "\n",
        "print(\"原始数据:\")\n",
        "print(df)\n",
        "\n",
        "# 创建缺失值指示变量\n",
        "df['年龄_缺失'] = df['年龄'].isna().astype(int)\n",
        "print(\"\\n添加缺失值指示变量:\")\n",
        "print(df)\n",
        "\n",
        "# 填充缺失值\n",
        "df['年龄'] = df['年龄'].fillna(df['年龄'].mean())\n",
        "print(\"\\n填充后:\")\n",
        "print(df)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 技巧3: 使用 apply 自定义填充逻辑"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "df = pd.DataFrame({\n",
        "    '姓名': ['张三', '李四', '王五', '赵六'],\n",
        "    '年龄': [25, 30, np.nan, 28],\n",
        "    '经验': ['新手', '老手', np.nan, '新手']\n",
        "})\n",
        "\n",
        "print(\"原始数据:\")\n",
        "print(df)\n",
        "\n",
        "def custom_fill(series):\n",
        "    if series.dtype == 'float64':\n",
        "        return series.fillna(series.mean())\n",
        "    else:\n",
        "        return series.fillna('未知')\n",
        "\n",
        "df = df.apply(custom_fill)\n",
        "print(\"\\n自定义填充后:\")\n",
        "print(df)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 总结\n",
        "\n",
        "### 方法对比\n",
        "\n",
        "| 方法 | 适用场景 | 优点 | 缺点 |\n",
        "|------|----------|------|------|\n",
        "| `dropna()` | 缺失值少或无法填充 | 简单直接 | 损失数据 |\n",
        "| `fillna(常数)` | 常数填充合理 | 简单 | 可能引入偏差 |\n",
        "| `fillna(统计值)` | 数值型数据 | 保持统计特性 | 忽略相关性 |\n",
        "| `fillna(method='ffill/bfill')` | 时间序列 | 保持趋势 | 可能有偏差 |\n",
        "| `interpolate()` | 有序数据 | 更精确 | 计算较慢 |\n",
        "\n",
        "### 关键要点\n",
        "\n",
        "1. **先检测后处理**：使用 `isna()` 了解缺失情况\n",
        "2. **根据业务选择**：删除 vs 填充需要结合业务逻辑\n",
        "3. **优先使用统计值**：均值、中位数、众数比常数更合理\n",
        "4. **时间序列用插值**：`interpolate()` 比简单填充更精确\n",
        "5. **记录处理过程**：创建缺失指示变量，保留信息\n",
        "6. **验证处理效果**：处理前后对比，确保合理"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "ml311",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}
